ci: publish CI images to internal registry first#4002
Draft
realFlowControl wants to merge 8 commits into
Draft
Conversation
|
realFlowControl
changed the title
- Update .gitlab/ci-images.yml to change the default CI_REGISTRY to registry.ddbuild.io and target the ddbuild registry path registry.ddbuild.io/ci/dd-trace-php/dd-trace-ci. - Make docker logins dynamic to support local builds, Docker Hub logins, and AWS ECR logins depending on the target registry server. - Bypass runner credential helper issues in Linux container environments by resetting ~/.docker/config.json. - Make registry and base image names fully configurable in docker-compose.yml and Dockerfiles, allowing parent base images to be dynamically resolved from ddbuild during child compilation steps.
- Update all GitLab CI generator scripts (.gitlab/generate-*.php) to use internal CI images from registry.ddbuild.io/ci/dd-trace-php/dd-trace-ci instead of pulling from Docker Hub via the mirror path. - This ensures test jobs use the newly compiled images directly from our project's ECR registry namespace.
- Add a new 'ci-publish' stage to .gitlab-ci.yml. - Implement 4 parallel matrix trigger jobs in .gitlab/ci-images.yml (Publish CentOS, Publish Bookworm, Publish Alpine, and Publish Windows) to run automatically after their respective build jobs succeed. - Each trigger calls the DataDog/public-images pipeline, passing the corresponding internal ddbuild ECR image as source and targeting public Docker Hub as destination under the exact same tag.
- Update all occurrences of bookworm-8 and shared-ext-8 to bookworm-9 and shared-ext-9 globally across .gitlab CI test generators, .gitlab/ci-images.yml, and .github workflows. - Update BOOKWORM_VERSION from 8 to 9 in tooling/bin/build-debug-artifact to ensure local debug builds pull and compile with the new version.
- Export MAKEFLAGS=-j at the top of build-extensions.sh. - This forces all underlying make invocations triggered by pecl install (including the heavy single-threaded gRPC, MongoDB, and parallel builds) to compile in parallel, drastically reducing build times on multi-core runner environments.
- Remove obsolete CI_REGISTRY, CI_REGISTRY_USER, and CI_REGISTRY_TOKEN from .gitlab/ci-images.yml. - Remove all complex, dynamic ECR/Docker Hub login shell blocks and AWS CLI installations from CentOS, Alpine, Bookworm, and Windows build jobs. - Rely entirely on the runner's native, pre-configured credentials for registry.ddbuild.io, significantly simplifying the pipeline configuration.
- Clean up dockerfiles/ci/README.md to document the new automated, secure internal ECR build flow. - Clarify that project collaborators no longer need to configure Personal Access Tokens (PATs) or credentials when building CI images. - Document how to trigger the manual sync to the public Docker Hub registry via downstream triggers in the 'ci-publish' stage.
realFlowControl
force-pushed
the
florian/ci-images
branch
8 times, most recently
from
6d45d97 to
93e1c93
Compare
GitLab cannot expand variables in needs:, so a single parallel manifest job
cannot depend on only its own version's two arch builds. Generate explicit
per-image jobs instead.
.gitlab/generate-ci-images.php reads the docker-compose.yml + .env files
(single source of truth for PHP versions and tags) and renders
.gitlab/ci-images.yml.tpl, emitting per Linux image:
- <OS> build : version x arch build matrix (compose build, no buildx
builder instance, then docker push the per-arch tag)
- <OS> manifest:<v>: auto-triggers (on_success) once its two arch builds
finish; needs them by literal name
- <OS> publish:<v> : manual, triggers DataDog/public-images (dockerhub only)
Static preamble + Windows jobs live in .gitlab/ci-images.static.yml (Windows
is single-arch, no manifest). The generator runs in generate-templates and is
triggered as a child pipeline via the new manual 'ci-images' job; the old
.gitlab/ci-images.yml local include is removed.
realFlowControl
force-pushed
the
florian/ci-images
branch
from
93e1c93 to
0464d2c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When building CI docker images, this PR changes the process to:
registry.ddbuild.io(Datadog internal container registry)public-imagesdownstream job to magically sync those images to Docker Hub for usage with GitHub CI and external contributorsWins
Reviewer checklist